{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Flowers Image Classification with TensorFlow on Cloud ML Engine TPU\n",
    "\n",
    "This notebook demonstrates how to do image classification from scratch on a flowers dataset using the Estimator API. Unlike [flowers_fromscratch.ipynb](the flowers_fromscratch notebook), here we do it on a TPU.\n",
    "\n",
    "Therefore, this will work only if you have quota for TPUs (not in Google Skills). It will cost about $3 if you want to try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install apache-beam[gcp]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "After doing a pip install, click on Reset Session so that the Python environment picks up the new package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "PROJECT = 'cloud-training-demos' # REPLACE WITH YOUR PROJECT ID\n",
    "BUCKET = 'cloud-training-demos-ml' # REPLACE WITH YOUR BUCKET NAME\n",
    "REGION = 'us-central1' # REPLACE WITH YOUR BUCKET REGION e.g. us-central1\n",
    "MODEL_TYPE = 'tpu'\n",
    "\n",
    "# do not change these\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['REGION'] = REGION\n",
    "os.environ['MODEL_TYPE'] = MODEL_TYPE\n",
    "os.environ['TFVERSION'] = '1.8'  # Tensorflow version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n",
      "Updated property [compute/region].\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Preprocess JPEG images to TF Records\n",
    "\n",
    "While using a GPU, it is okay to read the JPEGS directly from our input_fn. However, TPUs are too fast and it will be very wasteful to have the TPUs wait on I/O. Therefore, we'll preprocess the JPEGs into TF Records.\n",
    "\n",
    "This runs on Cloud Dataflow and will take <b> 15-20 minutes </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud storage cat gs://cloud-ml-data/img/flower_photos/train_set.csv  | sed 's/,/ /g' | awk '{print $2}' | sort | uniq > /tmp/labels.txt"   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3300\n",
      "370\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud storage cat gs://cloud-ml-data/img/flower_photos/train_set.csv | wc -l\n",    "gcloud storage cat gs://cloud-ml-data/img/flower_photos/eval_set.csv | wc -l"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "export PYTHONPATH=${PYTHONPATH}:${PWD}/flowersmodeltpu\n",
    "gcloud storage rm --recursive --continue-on-error gs://${BUCKET}/tpu/flowers/data\n",    "python -m trainer.preprocess \\\n",
    "       --train_csv gs://cloud-ml-data/img/flower_photos/train_set.csv \\\n",
    "       --validation_csv gs://cloud-ml-data/img/flower_photos/eval_set.csv \\\n",
    "       --labels_file /tmp/labels.txt \\\n",
    "       --project_id $PROJECT \\\n",
    "       --output_dir gs://${BUCKET}/tpu/flowers/data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00000-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00001-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00002-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00003-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00004-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00005-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00006-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/train-00007-of-00008\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/validation-00000-of-00003\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/validation-00001-of-00003\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/validation-00002-of-00003\n",
      "gs://cloud-training-demos-ml/tpu/flowers/data/tmp/\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud storage ls gs://${BUCKET}/tpu/flowers/data/"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Run as a Python module\n",
    "\n",
    "First run locally without --use_tpu -- don't be concerned if the process gets killed for using too much memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "WITHOUT_TPU=\"--train_batch_size=2  --train_steps=5\"\n",
    "OUTDIR=./flowers_trained\n",
    "rm -rf $OUTDIR\n",
    "export PYTHONPATH=${PYTHONPATH}:${PWD}/flowersmodeltpu\n",
    "python -m flowersmodeltpu.task \\\n",
    "   --output_dir=$OUTDIR \\\n",
    "   --num_train_images=3300 \\\n",
    "   --num_eval_images=370 \\\n",
    "   $WITHOUT_TPU \\\n",
    "   --learning_rate=0.01 \\\n",
    "   --project=${PROJECT} \\\n",
    "   --train_data_path=gs://${BUCKET}/tpu/flowers/data/train* \\\n",
    "   --eval_data_path=gs://${BUCKET}/tpu/flowers/data/validation*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then, run it on Cloud ML Engine with --use_tpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://cloud-training-demos-ml/flowers/trained_tpu_delete us-central1 flowers_tpu_180827_182402\n",
      "jobId: flowers_tpu_180827_182402\n",
      "state: QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "CommandException: 1 files/objects could not be removed.\n",
      "Job [flowers_tpu_180827_182402] submitted successfully.\n",
      "Your job is still active. You may view the status of your job with the command\n",
      "\n",
      "  $ gcloud ml-engine jobs describe flowers_tpu_180827_182402\n",
      "\n",
      "or continue streaming the logs with the command\n",
      "\n",
      "  $ gcloud ml-engine jobs stream-logs flowers_tpu_180827_182402\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "WITH_TPU=\"--train_batch_size=256  --train_steps=3000 --batch_norm --use_tpu\"\n",
    "WITHOUT_TPU=\"--train_batch_size=2  --train_steps=5\"\n",
    "OUTDIR=gs://${BUCKET}/flowers/trained_${MODEL_TYPE}_delete\n",
    "JOBNAME=flowers_${MODEL_TYPE}_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gcloud storage rm --recursive --continue-on-error $OUTDIR\n",
    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=flowersmodeltpu.task \\\n",
    "   --package-path=${PWD}/flowersmodeltpu \\\n",
    "   --job-dir=$OUTDIR \\\n",
    "   --staging-bucket=gs://$BUCKET \\\n",
    "   --scale-tier=BASIC_TPU \\\n",
    "   --runtime-version=$TFVERSION \\\n",
    "   -- \\\n",
    "   --output_dir=$OUTDIR \\\n",
    "   --num_train_images=3300 \\\n",
    "   --num_eval_images=370 \\\n",
    "   $WITH_TPU \\\n",
    "   --learning_rate=0.01 \\\n",
    "   --project=${PROJECT} \\\n",
    "   --train_data_path=gs://${BUCKET}/tpu/flowers/data/train-* \\\n",
    "   --eval_data_path=gs://${BUCKET}/tpu/flowers/data/validation-*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n",
      "\n",
      "signature_def['classes']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "    inputs['image_bytes'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: ()\n",
      "        name: Placeholder:0\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['class'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (1)\n",
      "        name: GatherV2:0\n",
      "    outputs['classid'] tensor_info:\n",
      "        dtype: DT_INT32\n",
      "        shape: (1)\n",
      "        name: Cast_1:0\n",
      "    outputs['probabilities'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (1, 5)\n",
      "        name: Softmax:0\n",
      "  Method name is: tensorflow/serving/predict\n",
      "\n",
      "signature_def['serving_default']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "    inputs['image_bytes'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: ()\n",
      "        name: Placeholder:0\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['class'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (1)\n",
      "        name: GatherV2:0\n",
      "    outputs['classid'] tensor_info:\n",
      "        dtype: DT_INT32\n",
      "        shape: (1)\n",
      "        name: Cast_1:0\n",
      "    outputs['probabilities'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (1, 5)\n",
      "        name: Softmax:0\n",
      "  Method name is: tensorflow/serving/predict\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/envs/py2env/lib/python2.7/site-packages/h5py/__init__.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._conv import register_converters as _register_converters\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/h5py/__init__.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from . import h5a, h5d, h5ds, h5f, h5fd, h5g, h5r, h5s, h5t, h5p, h5z\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/h5py/_hl/group.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .. import h5g, h5i, h5o, h5r, h5t, h5l, h5p\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/sparse/lil.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from . import _csparsetools\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/sparse/csgraph/__init__.py:165: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._shortest_path import shortest_path, floyd_warshall, dijkstra,\\\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/sparse/csgraph/_validation.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._tools import csgraph_to_dense, csgraph_from_dense,\\\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/sparse/csgraph/__init__.py:167: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._traversal import breadth_first_order, depth_first_order, \\\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/sparse/csgraph/__init__.py:169: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._min_spanning_tree import minimum_spanning_tree\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/sparse/csgraph/__init__.py:170: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._reordering import reverse_cuthill_mckee, maximum_bipartite_matching, \\\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/linalg/basic.py:17: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._solve_toeplitz import levinson\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/linalg/__init__.py:207: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._decomp_update import *\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/special/__init__.py:640: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._ufuncs import *\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/special/_ellip_harm.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._ellip_harm_2 import _ellipsoid, _ellipsoid_norm\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/interpolate/_bsplines.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from . import _bspl\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/spatial/__init__.py:95: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .ckdtree import *\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/spatial/__init__.py:96: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .qhull import *\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/spatial/_spherical_voronoi.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from . import _voronoi\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/spatial/distance.py:122: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from . import _hausdorff\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/ndimage/measurements.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from . import _ni_label\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/_libs/__init__.py:4: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/__init__.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs import (hashtable as _hashtable,\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/dtypes/common.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs import algos, lib\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/util/hashing.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs import hashing, tslib\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/indexes/base.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs import (lib, index as libindex, tslib as libts,\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/tools/datetimes.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs.tslibs.strptime import array_strptime\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/tseries/frequencies.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs.tslibs.frequencies import (  # noqa\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/indexes/datetimelike.py:28: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs.period import Period\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/sparse/array.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  import pandas._libs.sparse as splib\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/window.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  import pandas._libs.window as _window\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/groupby.py:68: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs import lib, groupby as libgroupby, Timestamp, NaT, iNaT\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/core/reshape/reshape.py:31: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from pandas._libs import algos as _algos, reshape as _reshape\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/pandas/io/parsers.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  import pandas._libs.parsers as parsers\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/sklearn/utils/__init__.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .murmurhash import murmurhash3_32\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/optimize/_trlib/__init__.py:1: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._trlib import TRLIBQuadraticSubproblem\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/optimize/_numdiff.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._group_columns import group_dense, group_sparse\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/scipy/stats/_continuous_distns.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from . import _stats\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/sklearn/utils/extmath.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from ._logistic_sigmoid import _log_logistic_sigmoid\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/sklearn/utils/extmath.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .sparsefuncs_fast import csr_row_norms\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/sklearn/metrics/cluster/supervised.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .expected_mutual_info_fast import expected_mutual_information\n",
      "/usr/local/envs/py2env/lib/python2.7/site-packages/sklearn/metrics/pairwise.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  from .pairwise_fast import _chi2_kernel_fast, _sparse_manhattan\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "MODEL_LOCATION=$(gcloud storage ls gs://${BUCKET}/flowers/trained_${MODEL_TYPE}/export/exporter | tail -1)\n",    "saved_model_cli show --dir $MODEL_LOCATION --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Monitoring training with TensorBoard\n",
    "\n",
    "Use this cell to launch tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from google.datalab.ml import TensorBoard\n",
    "TensorBoard().start('gs://{}/flowers/trained_{}'.format(BUCKET, MODEL_TYPE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "for pid in TensorBoard.list()['pid']:\n",
    "  TensorBoard().stop(pid)\n",
    "  print 'Stopped TensorBoard with pid {}'.format(pid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Deploying and predicting with model\n",
    "\n",
    "Deploy the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting and deploying flowers tpu from gs://cloud-training-demos-ml/flowers/trained_tpu/export/exporter/1534800612/ ... this will take a few minutes\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating version (this might take a few minutes)......\n",
      "........................................................................................................done.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "MODEL_NAME=\"flowers\"\n",
    "MODEL_VERSION=${MODEL_TYPE}\n",
    "MODEL_LOCATION=$(gcloud storage ls gs://${BUCKET}/flowers/trained_${MODEL_TYPE}/export/exporter | tail -1)\n",    "echo \"Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes\"\n",
    "#gcloud ml-engine versions delete --quiet ${MODEL_VERSION} --model ${MODEL_NAME}\n",
    "#gcloud ml-engine models delete ${MODEL_NAME}\n",
    "#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION\n",
    "gcloud alpha ml-engine versions create ${MODEL_VERSION} --machine-type mls1-c4-m4 --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version=$TFVERSION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To predict with the model, let's take one of the example images that is available on Google Cloud Storage <img src=\"http://storage.googleapis.com/cloud-ml-data/img/flower_photos/sunflowers/1022552002_2b93faf9e7_n.jpg\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud alpha ml-engine models list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The online prediction service expects images to be base64 encoded as described [here](https://cloud.google.com/ml-engine/docs/tensorflow/online-predict#binary_data_in_prediction_input)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "IMAGE_URL=gs://cloud-ml-data/img/flower_photos/sunflowers/1022552002_2b93faf9e7_n.jpg\n",
    "\n",
    "# Copy the image to local disk.\n",
    "gcloud storage cp $IMAGE_URL flower.jpg\n",    "\n",
    "# Base64 encode and create request message in json format.\n",
    "python -c 'import base64, sys, json; img = base64.b64encode(open(\"flower.jpg\", \"rb\").read()).decode(); print(json.dumps({\"image_bytes\":{\"b64\": img}}))' &> request.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Send it to the prediction service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: (gcloud.ml-engine.predict) HTTP request failed. Response: {\n",
      "  \"error\": {\n",
      "    \"code\": 500,\n",
      "    \"message\": \"Internal error encountered.\",\n",
      "    \"status\": \"INTERNAL\",\n",
      "    \"details\": [\n",
      "      {\n",
      "        \"@type\": \"type.googleapis.com/google.rpc.DebugInfo\",\n",
      "        \"detail\": \"[ORIGINAL ERROR] generic::internal: Prediction failed.\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud ml-engine predict \\\n",
    "  --model=flowers2 \\\n",
    "  --version=${MODEL_TYPE} \\\n",
    "  --json-instances=./request.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<pre>\n",
    "# Copyright 2017 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "</pre>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
